Vehicle control device, vehicle control method, and storage medium

ABSTRACT

Provided is a vehicle control device configured to: recognize a surrounding situation of a vehicle; control steering and acceleration/deceleration of the vehicle; detect operation states of a plurality of external recognition sensors; determine a driving mode of the vehicle as any one of a plurality of driving modes including a first driving mode, a second driving mode, and a third driving mode; change the third driving mode to either one of the first driving mode and the second driving mode when determining that a failure has occurred in one of the plurality of external recognition sensors that implement: a function of causing the vehicle to follow a preceding vehicle; and a function of assisting the vehicle in keeping a lane; and change the third driving mode to the second driving mode when determining that degradation in performance has occurred in one of the plurality of external recognition sensors.

CROSS-REFERENCE TO RELATED APPLICATION

The application is based on Japanese Patent Application No. 2020-218697filed on Dec. 28, 2020, the content of which incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to a vehicle control device, a vehiclecontrol method, and a storage medium.

Description of Related Art

In a vehicle control system that can handle automatic driving, thetechnology of safely evacuating and stopping a vehicle when a driverdoes not perform an operation of driving the vehicle is known. Forexample, Japanese Patent Application Laid-open No. 2020-166667 disclosesthe technology of executing the processing of stopping a vehicle withinan allowable period when a predetermined condition indicating thatcontinuation of driving of the vehicle by a driver is difficult issatisfied.

SUMMARY

However, the technology described in Japanese Patent ApplicationLaid-open No. 2020-166667 relates to control at a time when continuationof driving of the vehicle by a driver is difficult, and does notconsider a case in which continuation of driving of the vehicle becomesdifficult due to occurrence of an abnormality in a sensor required forautomatic driving. As a result, appropriate control cannot be performedin some cases when an abnormality has occurred in a sensor required forautomatic driving.

The present invention has been made in view of the above-mentionedcircumstances, and has an object to provide a vehicle control device, avehicle control method, and a storage medium, which are capable ofperforming appropriate control when an abnormality has occurred in asensor required for automatic driving.

A vehicle control device according to the present invention adopts thefollowing configuration.

According to the aspects of (1) to (6), it is possible to performappropriate control when an abnormality has occurred in a sensorrequired for automatic driving.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a configuration diagram of a vehicle system that uses avehicle control device according to a first embodiment.

FIG. 2 is a functional configuration diagram of a first controller and asecond controller.

FIG. 3 is a diagram illustrating an example of a correspondencerelationship among a driving mode, a control state of an own vehicle,and a task.

FIG. 4 is a diagram illustrating processing to be executed by a modedeterminer when a failure has occurred in only the LIDAR.

FIG. 5 is a flow chart illustrating an example of a flow of processingto be executed by a recognizer and the mode determiner according to thefirst embodiment.

FIG. 6 is a flow chart illustrating an example of a flow of processingto be executed by a recognizer and a mode determiner according to asecond embodiment.

DESCRIPTION OF EMBODIMENTS

Now, description is given of a vehicle control device, a vehicle controlmethod, and a storage medium according to embodiments of the presentinvention with reference to the drawings.

First Embodiment [Overall Configuration]

FIG. 1 is a configuration diagram of a vehicle system 1 that uses avehicle control device according to a first embodiment. A vehicleincluding the vehicle system 1 is, for example, a vehicle such as atwo-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle,and its power source is an internal combustion engine such as a dieselengine or a gasoline engine, an electric motor, or a combinationthereof. The electric motor operates by using power generated by agenerator connected to the internal combustion engine or powerdischarged by a secondary battery or a fuel cell.

The vehicle system 1 includes, for example, a camera 10, a radar device12, a LIDAR (Light Detection and Ranging) device 14, an objectrecognition device 16, a communication device 20, an HMI (Human MachineInterface) 30, a vehicle sensor 40, a navigation device 50, an MPU (MapPositioning Unit) 60, a driver monitoring camera 70, a driving operator80, an automatic driving control device 100, a driving force outputdevice 200, a braking device 210, and a steering device 220. Thesedevices and instruments are connected to one another via, for example, awireless communication line, a serial communication line, or a multiplexcommunication line such as a CAN (Controller Area Network) communicationline. The configuration illustrated in FIG. 1 is only one example, and apart of the configuration may be omitted, or another configuration maybe added. The camera 10, the radar device 12, and the LIDAR 14 arehereinafter sometimes referred to as “external recognition sensor”.

The camera 10 is, for example, a digital camera that uses a solid imagepickup device such as a CCD (Charge Coupled Device) or a CMOS(Complementary Metal Oxide Semiconductor). The camera 10 is mounted onany part of a vehicle (hereinafter referred to as “own vehicle M”)including the vehicle system 1. When the camera 10 picks up a frontimage, the camera 10 is mounted on, for example, an upper part of afront windshield or a back surface of a rear-view mirror. The camera 10repeatedly photographs the surroundings of the own vehicle Mperiodically, for example. The camera 10 may be a stereo camera.

The radar device 12 radiates a radio wave such as a millimeter wavetoward the surroundings of the own vehicle M, and detects a radio wave(reflected wave) reflected by an object, to detect at least the position(distance and direction) of the object. The radar device 12 is mountedon any part of the own vehicle M. The radar device 12 may detect theposition and speed of the object by an FM-CW (Frequency ModulatedContinuous Wave) method.

The LIDAR 14 radiates light (or electromagnetic wave having a wavelengthclose to light) toward the surroundings of the own vehicle M, andmeasures diffused light. The LIDAR 14 detects a distance to a targetbased on a period of time since emission of light until reception oflight. The light to be radiated is, for example, pulsed laser light. TheLIDAR 14 is mounted on any part of the own vehicle M.

The object recognition device 16 executes sensor fusion processing forresults of detection by a part or all of the camera 10, the radar device12, and the LIDAR 14, to thereby recognize a position, a type, and aspeed of an object, for example. The object recognition device 16outputs the recognition result to the automatic driving control device100. The object recognition device 16 may output the results ofdetection by the camera 10, the radar device 12, and the LIDAR 14 to theautomatic driving control device 100 as they are. The object recognitiondevice 16 may be omitted from the vehicle system 1. The objectrecognition device 16 further recognizes the operation states of thecamera 10, the radar device 12, and the LIDAR 14, and transmits therecognized operation states to the recognizer 130.

The communication device 20 uses, for example, a cellular network, aWi-Fi network, Bluetooth (trademark), or DSRC (Dedicated Short RangeCommunication) to communicate with another vehicle existing near the ownvehicle M or communicate with various kinds of server devices via aradio base station.

The HMI 30 presents various kinds of information to an occupant of theown vehicle M, and receives input of an operation by the occupant. TheHMI 30 includes, for example, various kinds of display devices,speakers, buzzers, touch panels, switches, and keys.

The vehicle sensor 40 includes, for example, a vehicle speed sensor thatdetects a speed of the own vehicle M, an acceleration sensor thatdetects an acceleration, a yaw rate sensor that detects an angular speedwith respect to a vertical axis, and an orientation sensor that detectsan orientation of the own vehicle M.

The navigation device 50 includes, for example, a GNSS (GlobalNavigation Satellite System) receiver 51, a navigation HMI 52, and aroute determiner 53. The navigation device 50 holds first mapinformation 54 in a storage device such as an HDD (Hard Disk Drive) or aflash memory. The GNSS receiver 51 identifies the position of the ownvehicle M based on a signal received from a GNSS satellite. The positionof the own vehicle M may be identified or complemented by an INS(Inertial Navigation System) that uses output of the vehicle sensor 40.The navigation HMI 52 includes, for example, a display device, aspeaker, a touch panel, and a key. The navigation HMI 52 and the HMI 30described above may be integrated partially or completely. The routedeterminer 53 refers to the first map information 54 to determine aroute (hereinafter referred to as “map route”) from the position (or anyinput position) of the own vehicle M identified by the GNSS receiver 51to a destination input by an occupant by using the navigation HMI 52,for example. The first map information 54 is, for example, informationrepresenting road structure by a link indicating a road and nodesconnected by the link. The first map information 54 may include, forexample, a curvature of a road and POI (Point Of Interest) information.The map route is output to the MPU 60. The navigation device 50 mayguide a route by using the navigation HMI 52 based on the map route. Thenavigation device 50 may be implemented by, for example, the function ofa terminal device such as a smartphone or a tablet terminal held by theoccupant. The navigation device 50 may transmit the current position andthe destination to a navigation server via the communication device 20,and acquire a route similar to the map route from the navigation server.

The MPU 60 includes, for example, a recommended lane determiner 61, andholds second map information 62 in a storage device such as an HDD or aflash memory. The recommended lane determiner 61 divides the map routeprovided by the navigation device 50 into a plurality of blocks (forexample, at intervals of 100 [m] with respect to a vehicle traveldirection), and determines a recommended lane for each block withreference to the second map information 62. The recommended lanedeterminer 61 determines on which lane the own vehicle M is to travel.When there is a junction on a map route, the recommended lane determiner61 determines a recommended route so that the own vehicle M can travelon a route for efficiently entering the junction.

The second map information 62 is map information having higher precisionthan that of the first map information 54. The second map information 62includes, for example, information on the center of a lane orinformation on the boundary of a lane. The second map information 62 mayinclude, for example, road information, traffic regulation information,address information (address or postal code), facility information,phone number information, and information on a section in which a mode Aor mode B described later is prohibited. The second map information 62may be updated appropriately through communication between thecommunication device 20 and another device.

A driver monitor camera 70 is, for example, a digital camera that uses asolid-state image pickup device such as a CCD image sensor or a CMOSimage sensor. The driver monitor camera 70 is mounted on any part of theown vehicle M at a position and in a direction so as to be capable ofpicking up a front image of a head of an occupant (hereinafter referredto as “driver”) sitting on a driver seat of the own vehicle M (in thedirection of picking up an image of the face). For example, the drivermonitor camera 70 is mounted on an upper part of a display deviceprovided on the center of an instrumental panel of the own vehicle M.

The driving operator 80 includes, for example, an acceleration pedal, abrake pedal, a gear shift, and other operators in addition to thesteering wheel 82. A sensor that detects an operation amount or whetheran operation is performed is mounted on the driving operator 80, and thedetection result is output to the automatic driving control device 100or a part or all of the driving force output device 200, the brakingdevice 210, and the steering device 220. The steering wheel 82 is anexample of an “operator that receives a steering operation performed bya driver”. The operator is not always required to have a ring shape, andmay have other shapes for steering, or may be a joystick or a button. Asteering grasp sensor 84 is attached to the steering wheel 82. Thesteering grasp sensor 84 is implemented by, for example, a capacitivesensor, and outputs, to the automatic driving control device 100, asignal that enables detection of whether or not the driver is graspingthe steering wheel 82 (in contact with the steering wheel 82 so as to beable to apply a force).

The automatic driving control device 100 includes, for example, a firstcontroller 120 and a second controller 160. The first controller 120 andthe second controller 160 are each implemented by a hardware processorsuch as a CPU (Central Processing Unit) executing a program (software).A part or all of the components may be implemented by hardware (circuit;including circuitry) such as an LSI (Large Scale Integration), an ASIC(Application Specific Integrated Circuit), an FPGA (Field-ProgrammableGate Array), or a GPU (Graphics Processing Unit), or may be implementedby cooperation between software and hardware. The program may be storedin advance in a storage device (storage device including anon-transitory storage medium) of the automatic driving control device100 such as an HDD or a flash memory, or the program may be stored in aremovable storage medium such as a DVD or a CD-ROM. Then, the storagemedium (non-transitory storage medium) may be attached to a drive deviceso that the program is installed into an HDD or a flash memory of theautomatic driving control device 100. The automatic driving controldevice 100 is an example of “vehicle control device”, and a combinationof an action plan generator 140 and a second controller 160 is anexample of “drive controller”.

FIG. 2 is a functional configuration diagram of the first controller 120and the second controller 160. The first controller 120 includes, forexample, a recognizer 130, an action plan generator 140, and a modedeterminer 150. The first controller 120 implements, for example, thefunction of AI (Artificial Intelligence) and the function of a modelgiven in advance in parallel. For example, the function of “recognizingan intersection” may be implemented by executing recognition of anintersection by, for example, deep learning, and recognition based on acondition (including, for example, a signal adapted for pattern matchingand a road sign) given in advance in parallel, giving scores to both ofthe recognitions, and giving an integral evaluation. In this manner, thereliability of automatic driving is ensured.

The recognizer 130 recognizes states such as the position, speed, andacceleration of an object near the own vehicle M based on informationinput from the camera 10, the radar device 12, and the LIDAR 14 via theobject recognition device 16. The position of an object is, for example,recognized as a position in an absolute coordinate system with respectto a representative point (for example, center of gravity or center ofdrive axis) of the own vehicle M to be used for control. The position ofan object may be represented by a representative point such as a centerof gravity or corner of the object, or may be represented by a region.The “state” of an object may include the acceleration, jerk, or “actionstate” (for example, whether or not the own vehicle M is changing a laneor is trying to change a lane) of the object.

The recognizer 130 recognizes, for example, a lane (traveling lane) onwhich the own vehicle M is traveling. For example, the recognizer 130recognizes the traveling lane by comparing a pattern (for example,arrangement of solid lines and broken lines) of a road division lineobtained from the second map information 62 with a pattern of a roaddivision line near the own vehicle M recognized from the imagephotographed by the camera 10, to thereby recognize the traveling lane.In addition to the road division line, the recognizer 130 may recognizethe traveling lane by recognizing a traveling path boundary (roadboundary) including, for example, a road division line, the shoulder ofa road, a curb, a center median, and a guardrail. The traveling lane maybe recognized in consideration of the position of the own vehicle Macquired from the navigation device 50 or the result of processing bythe INS. The recognizer 130 recognizes a stop line, an obstacle, redlight, a toll gate, and other road events. In this recognition, theposition of the own vehicle M acquired from the navigation device 50 orthe result of processing by an INS may be considered. Furthermore, therecognizer 130 recognizes a stop line, an obstacle, a red light, a tollgate, and other road events.

The recognizer 130 recognizes the position or posture of the own vehicleM with respect to a traveling lane when recognizing the traveling lane.The recognizer 130 may recognize, for example, as the relative positionand posture of the own vehicle M with respect to the traveling lane, adeviation of the reference point of the own vehicle M from the center ofthe lane and an angle with respect to a line obtained by connecting thecenters of the lane in the traveling direction of the own vehicle M.Instead, the recognizer 130 may recognize, for example, the position ofthe reference point of the own vehicle M with respect to any side edge(road division line or road boundary) of the traveling lane as therelative position of the own vehicle M with respect to the travelinglane.

Furthermore, the recognizer 130 acquires the operation state of theexternal recognition sensor from the object recognition device 16, anddetermines whether or not degradation in performance or a failure hasoccurred in the external recognition sensor. The specific method ofdetermining degradation in performance or a failure is described later.

The action plan generator 140 generates such a target trajectory thatthe own vehicle M travels in the future (irrespective of the operationof the driver) automatically on a recommended lane determined by therecommended lane determiner 61 in principle so as to be capable ofcoping with the surroundings situation of the own vehicle M. The targettrajectory includes, for example, a speed component. For example, thetarget trajectory is represented by arranging the locations (trajectorypoints) to be reached by the own vehicle M. The trajectory points arelocations to be reached by the own vehicle M at predetermined travelleddistances (for example, about several meters) along the road. Inaddition, a target speed and a target acceleration are generated in eachpredetermined sampling period (for example, less than 1 second) as apart of the target trajectory. The trajectory points may be positions tobe reached by the own vehicle M in each sampling period. In this case,information on the target speed and the target acceleration isrepresented by an interval between trajectory points.

The action plan generator 140 may set an automatic driving event whengenerating a target trajectory. The automatic driving event includes,for example, a constant speed traveling event, a low-speed followingtraveling event, a lane change event, a junction event, a merge event,and a takeover event. The action plan generator 140 generates a targettrajectory that depends on an activated event.

The mode determiner 150 determines a driving mode of the own vehicle Mas any one of a plurality of driving modes having different tasksimposed on a driver. The mode determiner 150 includes, for example, adriver state determiner 152 and a mode change processor 154. Thefunctions of these components are described later.

FIG. 3 is a diagram illustrating an example of a correspondencerelationship among a driving mode, a control state of the own vehicle M,and a task. The driving mode of the own vehicle M includes, for example,five modes, namely, a mode A to a mode E. The control state, namely, thedegree of automatic driving control of the own vehicle M is the highestfor the mode A, and the degree of automatic driving control decreases inorder of the mode B, the mode C, the mode D, and the mode E. Incontrast, the degree of a task imposed on a driver is the smallest forthe mode A, and increases in order of the mode B, the mode C, the modeD, and the mode E. The mode D or the mode E is a control state that isnot automatic driving, and thus the automatic driving control device 100has a responsibility to finish control relating to automatic driving,and cause the driving mode to transition to driving assistance or manualdriving. Now, examples of details of the respective driving modes aregiven in the following.

The mode A relates to the state of automatic driving, and the driverdoes not bear any one of the tasks of monitoring the surroundings of theown vehicle M and grasping the steering wheel 82. Monitoring of thesurroundings of the own vehicle M includes at least monitoring of thefront field of view of the own vehicle M. However, even in the mode A,the driver is required to have a posture of being able to immediatelytransition to manual driving in response to a request from a system,which is mainly the automatic driving control device 100. The automaticdriving indicates that both of steering and acceleration/decelerationare controlled irrespective of an operation of the driver. The frontfield of view means a space in the traveling direction of the ownvehicle M visually recognized through a front wind shield. The mode A isa driving mode that can be executed, for example, when the own vehicle Mis traveling at a speed equal to or lower than the upper limit vehiclespeed (for example, about 50 [km/h]) on an expressway such as a highway,and there is a preceding vehicle for the own vehicle M to follow, whichis sometimes referred to as TJP (Traffic Jam Pilot). When this conditionis not satisfied, the mode determiner 150 changes the driving mode ofthe own vehicle M to the mode B. The mode A and/or the mode B is anexample of “third driving mode”, the mode C is an example of “seconddriving mode”, and the mode D and/or the mode E is an example of “firstdriving mode”.

The mode B relates to the state of driving assistance, and the driverbears the task of monitoring the front field of view of the own vehicleM, but does not have the task of grasping the steering wheel 82. Themode C relates to the state of driving assistance, and the driver bearsthe task of monitoring the front field of view, and the task of graspingthe steering wheel 82. The mode D is a driving mode in which the driveris required to perform a certain degree of operation for at least one ofsteering and acceleration/deceleration of the own vehicle M. Forexample, in the mode C or the mode D, driving assistance such as ACC(Adaptive Cruise Control) or LKAS (Lane Keeping Assist System) isperformed. The ACC is a function of causing the own vehicle M to followa preceding vehicle while keeping a constant inter-vehicle distancebetween the own vehicle M and the preceding vehicle, and LKAS is afunction of assisting the own vehicle M in keeping the lane so that theown vehicle M travels along the center of the traveling lane. The mode Erefers to the state of manual driving in which the driver is required toperform a driving operation for both of steering andacceleration/deceleration, and driving assistance such as ACC or LKAS isnot performed. In both of the mode D and the mode E, the driver bearsthe task of monitoring the front field of view of the own vehicle M.

The automatic driving control device 100 (and driving assistance device(not shown)) executes automatic lane change that depends on the drivingmode. The automatic lane change includes an automatic lane change (1)required by the system and an automatic lane change (2) required by thedriver. The automatic lane change (1) includes an automatic lane changefor passing a preceding vehicle, which is performed when the vehiclespeed of the preceding vehicle is lower than the vehicle speed of theown vehicle by a reference amount or more, and an automatic lane change(automatic lane change caused by change of recommended lane) fortraveling toward the destination. The automatic lane change (2) is tochange the lane of the own vehicle M toward an operation direction whenthe driver has operated a blinker in a case where, for example, acondition on the vehicle speed or a positional relationship with anearby vehicle is satisfied.

In the mode A, the automatic driving control device 100 does not executeany one of the automatic lane change (1) and the automatic lane change(2). In the mode B and the mode C, the automatic driving control device100 executes both of the automatic lane change (1) and the automaticlane change (2). In the mode D, the driving assistance device (notshown) does not execute the automatic lane change (1) but executes theautomatic lane change (2). In the mode E, both of the automatic lanechange (1) and the automatic lane change (2) are not executed.

When the task of the determined driving mode (hereinafter referred to as“current driving mode”) is not performed by the driver, the modedeterminer 150 changes the driving mode of the own vehicle M to adriving mode that imposes a heavier task.

For example, when the driver is in a posture of not being able totransition to manual driving in response to a request from the system inthe mode A (for example, when the driver is continuously looking asideor when a sign that indicates a difficulty in driving is detected), themode determiner 150 uses the HMI 30 to prompt the driver to transitionto manual driving, and when the driver does not respond, the modedeterminer 150 performs control of causing the own vehicle M togradually stop at the shoulder of the road and stopping automaticdriving. After automatic driving is stopped, the own vehicle M is set tothe state of the mode D or the mode E, and the own vehicle M can becaused to start by a manual operation performed by the driver. Thefollowing description holds true for the case of “stopping automaticdriving”. When the driver is not monitoring the front field of view inthe mode B, the mode determiner 150 uses the HMI 30 to prompt the driverto monitor the front field of view, and when the driver does notrespond, the mode determiner 150 performs control of causing the ownvehicle M to gradually stop at the shoulder of the road and stoppingautomatic driving. When the driver is not monitoring the front field ofview or is not grasping the steering wheel 82 in the mode C, the modedeterminer 150 uses the HMI 30 to prompt the driver to monitor the frontfield of view and/or to grasp the steering wheel 82, and when the driverdoes not respond, the mode determiner 150 performs control of causingthe own vehicle M to gradually stop at the shoulder of the road andstopping automatic driving.

The driver state determiner 152 monitors the state of the driver anddetermines whether the state of the driver is a state that depends on atask in order to perform the mode change described above. For example,the driver state determiner 152 analyzes an image photographed by thedriver monitor camera 70 to perform posture estimation processing, anddetermines whether the driver is in a posture of not being able totransition to manual driving in response to a request from the system.The driver state determiner 152 analyzes the image photographed by thedriver monitor camera 70 to perform line-of-sight estimation processing,and determines whether or not the driver is monitoring the front fieldof view.

The mode change processor 154 performs various kinds of processing forchanging the mode. For example, the mode change processor 154 gives aninstruction to generate a target trajectory for causing the action plangenerator 140 to stop at the shoulder, gives an activation instructionto the driving assistance device (not shown), or controls the HMI 30 toprompt the driver to perform an action.

The second controller 160 controls the driving force output device 200,the braking device 210, and the steering device 220 so that the ownvehicle M passes through the target trajectory generated by the actionplan generator 140 as scheduled.

Referring back to FIG. 2, the second controller 160 includes, forexample, an acquirer 162, a speed controller 164, and a steeringcontroller 166. The acquirer 162 acquires information on a targettrajectory (trajectory points) generated by the action plan generator140, and stores the information into a memory (not shown). The speedcontroller 164 controls the driving force output device 200 or thebraking device 210 based on a speed component accompanying the targettrajectory stored in the memory. The steering controller 166 controlsthe steering device 220 depending on the degree of curve of the targettrajectory stored in the memory. The processing of the speed controller164 and the steering controller 166 is implemented by a combination offeed-forward control and feedback control, for example. As an example,the steering controller 166 executes feed-forward control that dependson the curvature of the road in front of the own vehicle M and feedbackcontrol based on a deviation from the target trajectory.

The driving force output device 200 outputs, to a drive wheel, atraveling driving force (torque) for causing the own vehicle M totravel. The driving force output device 200 includes, for example, acombination of an internal combustion engine, an electric motor, and atransmission, and an ECU (Electronic Control Unit) configured to controlthese components. The ECU controls the above-mentioned components inaccordance with information input from the second controller 160 orinformation input from the driving operator 80.

The braking device 210 includes, for example, a brake caliper, acylinder that transmits a hydraulic pressure to the brake caliper, anelectric motor that causes the hydraulic pressure in the cylinder, and abrake ECU. The brake ECU controls the electric motor in accordance withinformation input from the second controller 160 or information inputfrom the driving operator 80, and causes a brake torque that depends ona braking operation to be output to each wheel. The braking device 210may include, as a backup, a mechanism for transmitting the hydraulicpressure, which is caused by an operation of the brake pedal included inthe driving operator 80, to the cylinder via a master cylinder. Theconfiguration of the braking device 210 is not limited to theconfiguration described above, and the braking device 210 may be anelectronic hydraulic brake device configured to control an actuator inaccordance with information input from the second controller 160, andtransmit the hydraulic pressure of the master cylinder to the cylinder.

The steering device 220 includes, for example, a steering ECU and anelectric motor. The electric motor causes a force in a rack-and-pinionmechanism to change the orientation of a steered wheel. The steering ECUdrives the electric motor in accordance with information input from thesecond controller 160 or information input from the driving operator 80to change the orientation of the steered wheel.

[Control Executed at Time of Degradation in Performance or Failure ofExternal Recognition Sensor]

Now, description is given of control to be executed by the recognizer130 and the mode determiner 150 in cooperation at the time ofdegradation in performance or failure of the external recognitionsensor. This control is executed when the own vehicle M is traveling inany one of the mode A or the mode B.

As described above, the recognizer 130 determines, based on operationconditions of external recognition sensors including at least the camera10, the radar device 12, and the LIDAR 14, whether or not degradation inperformance or a failure has occurred in one of the external recognitionsensors. When the recognizer 130 has determined that degradation inperformance or a failure has occurred in one of the external recognitionsensors, the recognizer 130 notifies the mode determiner 150 ofinformation indicating that degradation in performance or a failure hasoccurred in one of the external recognition sensors. In the case of thecamera 10, for example, the recognizer 130 determines that performanceof the camera 10 is degraded when dirt adheres to the surface of thecamera 10, or determines that the camera 10 has failed when power is notsupplied to the camera 10 due to a failure in power adapter of thecamera 10. In the case of the radar device, for example, the recognizer130 determines that performance of the radar device 12 is degraded whena signal-to-noise ratio of the radar device 12 has become smaller thanusual, or determines that the radar device 12 has failed whentransmission of a radio wave has failed due to a failure in transmitterof the radar device 12. In the case of the LIDAR 14, for example, therecognizer 130 determines that performance of the LIDAR 14 is degradedwhen a foreign matter adheres to the surface of the LIDAR 14 andreflected light is immediately detected, or determines that the LIDAR 14has failed when a lens of the LIDAR 14 is broken.

When the mode determiner 150 has received, from the recognizer 130,information indicating that at least one of the external recognitionsensors is degraded, the mode determiner 150 changes the mode A or themode B to the mode C. That is, the driver bears a task of monitoring thefront field of view and a task of grasping the steering wheel 82. Thisis because, when at least one of the external recognition sensors isdegraded, for example, performance of avoiding an obstacle by automaticdriving is also degraded in some cases, and thus it is possible to causethe driver to be aware of decrease in level of automatic driving andcope with an abrupt problem by imposing the task of grasping thesteering wheel 82 on the driver. As a result, it is possible to performappropriate control when an abnormality has occurred in a sensorrequired for automatic driving.

When the mode determiner 150 has received from, the recognizer 130,information indicating that a failure has occurred in at least one ofthe external recognition sensors, the mode determiner 150 changes themode A or the mode B to the mode E. That is, automatic driving isswitched to manual driving in which the driver is required to performoperations for both of steering and acceleration/deceleration, anddriving assistance such as ACC or LKAS is not performed. This is becausewhen a failure has occurred in at least one of the external recognitionsensors, continuation of automatic driving in a state in which the taskof monitoring the front field of view or the task of grasping thesteering wheel 82 is not imposed, like the mode A or the mode B, has apossibility of threatening safety of the driver. It is possible tomotivate the driver to repair the external recognition sensor that hasfailed while at the same time ensuring safety of the driver by stoppingautomatic driving and imposing the task of manual driving on the driver.

Meanwhile, when the mode determiner 150 has received, from therecognizer 130, information indicating that a failure has occurred in atleast one of the external recognition sensors and the externalrecognition sensor that has failed is only the LIDAR 14, the modedeterminer 150 changes the mode A or the mode B to the mode C for apredetermined period, and then changes the mode C to the mode E. Drivingassistance such as ACC or LKAS may be performed under the state of themode C in this predetermined period. This is because when a failure hasoccurred in only the LIDAR 14, it is possible to continue driving in themode C temporarily by using the camera 10 and the radar device 12. As aresult, it is possible to prevent the driver from feeling strange due tothe driving mode being switched from the mode A or the mode B to themode E abruptly.

FIG. 4 is a diagram illustrating processing to be executed by the modedeterminer 150 when a failure has occurred in only the LIDAR. Asillustrated in FIG. 4, the own vehicle M is traveling in the mode Abefore a time point t0, and at the time point t0, the own vehicle Mreceives, from the recognizer 130, information indicating that a failurehas occurred in only the LIDAR 14 among the external recognitionsensors. At this time, the mode determiner 150 changes the driving modefrom the mode A to the mode C, and causes the driver to grasp thesteering wheel 82 while at the same time continuing driving in the modeC temporarily by using the camera 10 and the radar device 12. Then,after a predetermined period T has elapsed, at a time point t1, the modedeterminer 150 changes the driving mode from the mode C to the mode E,and imposes the task of manual driving on the driver. In this manner,when a failure has occurred in only the LIDAR 14, the mode determiner150 changes the driving mode to the mode C temporarily, and then to themode E. As a result, it is possible to ensure safety of the driverwithout causing the driver to feel strange.

In the example of FIG. 4, when a failure has occurred in only the LIDAR14, the mode determiner 150 sets the driving mode to the mode C for apredetermined period T. However, the configuration is not limitedthereto, and for example, the mode determiner 150 may set the drivingmode to the mode C until the own vehicle M has travelled for apredetermined distance, and after that, the mode determiner 150 maychange the driving mode to the mode E. Also with this configuration, itis possible to ensure safety of the driver without causing the driver tofeel strange.

Next, description is given of an example of processing to be executed bythe recognizer 130 and the mode determiner 150 when degradation inperformance or a failure has occurred in an external recognition sensor.FIG. 5 is a flow chart illustrating an example of a flow of processingto be executed by the recognizer 130 and the mode determiner 150according to the first embodiment.

First, the mode determiner 150 determines whether or not the currentdriving mode is the mode A or the mode B (Step S100). When the modedeterminer 150 has determined that the current driving mode is the modeA or the mode B, the recognizer 130 determines whether or not a failurehas occurred in at least one of the external recognition sensors (StepS101). When a failure has not occurred in at least one of the externalrecognition sensors, the recognizer 130 determines whether or notperformance of at least one of the external recognition sensors isdegraded (Step S102). When the recognizer 130 has determined thatperformance of at least one of the external recognition sensors is notdegraded, the recognizer 130 returns the processing to Step S101. On theother hand, when the recognizer 130 has determined that performance ofat least one of the external recognition sensors is degraded, the modedeterminer 150 changes the driving mode from the mode A or the mode B tothe mode C (Step S103).

When the recognizer 130 has determined that a failure has occurred in atleast one of the external recognition sensors, the recognizer 130 nextdetermines whether or not a failure has occurred in only the LIDAR 14(Step S104). When the recognizer 130 has determined that a failure hasoccurred in only the LIDAR 14, the mode determiner 150 sets the drivingmode to the mode C for a predetermined period, and after that, the modedeterminer 150 changes the mode C to the mode E (Step S105). On theother hand, when the recognizer 130 has determined that a failure hasnot occurred in only the LIDAR 14, the mode determiner 150 changes thedriving mode to the mode E (Step S106).

According to the first embodiment described above, the mode determiner150 changes the driving mode from the mode A or the mode B depending ondegradation in performance or a failure of the external recognitionsensor. Therefore, it is possible to perform appropriate control when anabnormality has occurred in a sensor required for automatic driving.

Second Embodiment

In the first embodiment described above, when it is determined thatperformance of at least one of the external recognition sensors isdegraded, the driving mode is changed to the mode C. In contrast, in asecond embodiment, even in a case where performance of at least one ofthe external recognition sensors is determined to be degraded, when theexternal recognition sensor whose performance is degraded is installedon the back side of the own vehicle M, driving in the mode A or the modeB is continued. This is because an external recognition sensor installedon the back side of the own vehicle M influences performance ofautomatic driving less than an external recognition sensor installed onthe front side of the own vehicle M.

FIG. 6 is a flow chart illustrating an example of a flow of processingto be executed by the recognizer 130 and the mode determiner 150according to the second embodiment.

First, the mode determiner 150 determines whether or not the currentdriving mode is the mode A or the mode B (Step S200). When the modedeterminer 150 has determined that the current driving mode is the modeA or the mode B, the recognizer 130 determines whether or not a failurehas occurred in at least one of the external recognition sensors (StepS201). When a failure has not occurred in at least one of the externalrecognition sensors, the recognizer 130 determines whether or notperformance of at least one of the external recognition sensors isdegraded (Step S202). When the recognizer 130 has determined thatperformance of at least one of the external recognition sensors is notdegraded, the recognizer 130 returns the processing to Step S101. On theother hand, when the recognizer 130 has determined that performance ofat least one of the external recognition sensors is degraded, therecognizer 130 determines whether the external recognition sensor whoseperformance is degraded is installed on the front side or lateral sideof the own vehicle M (Step S203). When the recognizer 130 has determinedthat the external recognition sensor whose performance is degraded isinstalled on the front side or lateral side of the own vehicle M, themode determiner 150 changes the driving mode from the mode A or the modeB to the mode C (Step S204). On the other hand, when the recognizer 130has determined that the external recognition sensor whose performance isdegraded is not installed on the front side or lateral side of the ownvehicle M, that is, when the recognizer 130 has determined that theexternal recognition sensor whose performance is degraded is notinstalled on the back side of the own vehicle M, the recognizer 130returns the processing to Step S201. The processing of from Step S205 toStep S207 is similar to the processing of from Step S104 to Step S106described above.

According to the second embodiment described above, when an externalrecognition sensor whose performance is degraded is installed on theback side of the own vehicle M, the mode determiner 150 keeps thedriving mode in the mode A or the mode B, and thus it is possible toperform more appropriate control without impairing the convenience forthe driver.

In the first embodiment and second embodiment described above, when itis determined that performance of an external recognition sensor isdegraded, the driving mode is changed to the mode C, or when it isdetermined that a failure has occurred in the external recognitionsensor, the driving mode is changed to the mode E. However, theconfiguration is not limited thereto, and the driving mode after changemay be a mode equal to or lower than the mode C.

Furthermore, the first embodiment and second embodiment described aboverelate to degradation in performance or a failure of the externalrecognition sensor. However, the processing of the present invention isnot solely directed to degradation in performance or a failure of theexternal recognition sensor, and can be applied to a sensor required forautomatic driving in general. For example, when a failure has occurredin an accelerator pedal position sensor mounted to an accelerator pedalof the driving operator 80, an operation performed by the driver cannotbe identified. In this case, the mode determiner 150 may not be able toidentify a driving state that depends on the state of the own vehicle M,and thus the mode determiner 150 may change the driving mode from themode A or the mode B to the mode C or a mode lower than the mode C.Furthermore, for example, when a redundant part forming a redundantarchitecture, which achieves the function of automatic driving, hasfailed, the mode determiner 150 may change the driving mode to the modeC or a mode lower than the mode C and cause the driver to grasp thesteering wheel 82 to prepare for an emergency.

This concludes the description of the embodiment for carrying out thepresent invention. The present invention is not limited to theembodiment in any manner, and various kinds of modifications andreplacements can be made within a range that does not depart from thegist of the present invention.

What is claimed is:
 1. A vehicle control device, comprising: one or morestorage devices storing a program; and one or more hardware processors,the one or more hardware processors executing the program stored in theone or more storage devices to: recognize a surrounding situation of avehicle; control steering and acceleration/deceleration of the vehiclebased on the recognized surrounding situation without depending on anoperation performed by a driver of the vehicle; detect operation statesof a plurality of external recognition sensors; determine a driving modeof the vehicle as any one of a plurality of driving modes including afirst driving mode, a second driving mode, and a third driving mode,wherein the third driving mode is a driving mode in which a load of atask imposed on the driver is smaller than a load of a task imposed inthe second driving mode, wherein the second driving mode is a drivingmode in which a load of a task imposed on the driver is smaller than aload of a task imposed in the first driving mode, wherein a part of theplurality of driving modes including at least the second driving modeand the third driving mode is performed by controlling steering andacceleration/deceleration of the vehicle without depending on anoperation performed by the driver of the vehicle, and wherein when atask in the determined driving mode is not performed by the driver, thedriving mode of the vehicle is changed to a driving mode imposing alarger task load; change the third driving mode to either one of thefirst driving mode and the second driving mode based on the operationstates when determining based on the operation states that a failure hasoccurred in at least one of the plurality of external recognitionsensors that implement: a function of causing the vehicle to follow apreceding vehicle while keeping a constant inter-vehicle distancebetween the vehicle and the preceding vehicle; and a function ofassisting the vehicle in keeping a lane so as to travel within a lane inwhich the vehicle is traveling; and change the third driving mode to thesecond driving mode when determining based on the operation states thatdegradation in performance, which is a phenomenon different from thefailure, has occurred in at least one of the plurality of externalrecognition sensors.
 2. A vehicle control device, comprising: one ormore storage devices storing a program; and one or more hardwareprocessors, the one or more hardware processors executing the programstored in the one or more storage devices to: recognize a surroundingsituation of a vehicle; control steering and acceleration/decelerationof the vehicle based on the recognized surrounding situation withoutdepending on an operation performed by a driver of the vehicle; detectoperation states of a plurality of external recognition sensors;determine a driving mode of the vehicle as any one of a plurality ofdriving modes including a first driving mode, a second driving mode, anda third driving mode, wherein the third driving mode is a driving modein which a load of a task imposed on the driver is smaller than a loadof a task imposed in the second driving mode, wherein the second drivingmode is a driving mode in which a load of a task imposed on the driveris smaller than a load of a task imposed in the first driving mode,wherein a part of the plurality of driving modes including at least thesecond driving mode and the third driving mode is performed bycontrolling steering and acceleration/deceleration of the vehiclewithout depending on an operation performed by the driver of thevehicle, and wherein when a task in the determined driving mode is notperformed by the driver, the driving mode of the vehicle is changed to adriving mode imposing a larger task load; change the third driving modeto either one of the first driving mode and the second driving modebased on the operation states when determining based on the operationstates that a failure has occurred in at least one of the plurality ofexternal recognition sensors; change the third driving mode to thesecond driving mode when determining based on the operation states thatdegradation in performance, which is a phenomenon different from thefailure, has occurred in at least one of the plurality of externalrecognition sensors; and continue the third driving mode whendetermining that a failure has occurred in at least one of the pluralityof external recognition sensors and the at least one of the plurality ofexternal recognition sensors that has failed is installed on a back sideof the vehicle.
 3. The vehicle control device according to claim 1,wherein the plurality of external recognition sensors include at least aradar, a camera, and a LIDAR, wherein, when determining that a failurehas occurred in only the LIDAR among the plurality of externalrecognition sensors, the one or more hardware processors set the thirddriving mode to the second driving mode for a predetermined period, andchange the second driving mode to the first driving mode aftercontinuing driving in the second driving mode by using the radar and thecamera, and wherein the one or more hardware processors change the thirddriving mode to the first driving mode when determining a failure hasoccurred in at least one of the radar and the camera.
 4. The vehiclecontrol device according to claim 1, wherein the plurality of externalrecognition sensors include at least a radar, a camera, and a LIDAR,wherein, when determining that a failure has occurred in only the LIDARamong the plurality of external recognition sensors, the one or morehardware processors set the third driving mode to the second drivingmode for a predetermined period, and change the second driving mode tothe first driving mode after enabling a function of causing the vehicleto follow a preceding vehicle while keeping a constant inter-vehicledistance between the vehicle and the preceding vehicle; and a functionof assisting the vehicle in keeping a lane so as to travel within a lanein which the vehicle is traveling.
 5. The vehicle control deviceaccording to claim 1, wherein the third driving mode is a driving modein which a task of grasping an operator that receives a steeringoperation is not imposed on the driver or a task of monitoring asurrounding situation is not imposed on the driver, wherein the seconddriving mode is a driving mode in which the task of monitoring asurrounding situation and the task of grasping an operator are imposedon the driver, and wherein the first driving mode is a driving mode inwhich a task of steering or accelerating/decelerating the vehicle isimposed on the driver.
 6. The vehicle control device according to claim1, wherein the degradation in performance is a phenomenon in whichperformance of at least one of the plurality of external recognitionsensors is degraded but is capable of recovered to normal performance.7. The vehicle control device according to claim 6, wherein the one ormore hardware processors change the driving mode from the second drivingmode to the third driving mode when determining that performance of atleast one of the plurality of external recognition sensors is degradedand then has recovered to the normal performance.
 8. The vehicle controldevice according to claim 6, wherein the one or more hardware processorschange the second driving mode to the third driving mode whendetermining that performance of at least one of the plurality ofexternal recognition sensors is degraded and then has recovered to thenormal performance after elapse of a predetermined period.
 9. A vehiclecontrol method to be executed by a computer mounted on a vehicle, thevehicle control method comprising: recognizing a surrounding situationof a vehicle; controlling steering and acceleration/deceleration of thevehicle based on the recognized surrounding situation without dependingon an operation performed by a driver of the vehicle; detectingoperation states of a plurality of external recognition sensors;determining a driving mode of the vehicle as any one of a plurality ofdriving modes including a first driving mode, a second driving mode, anda third driving mode, wherein the third driving mode is a driving modein which a load of a task imposed on the driver is smaller than a loadof a task imposed in the second driving mode, wherein the second drivingmode is a driving mode in which a load of a task imposed on the driveris smaller than a load of a task imposed in the first driving mode,wherein a part of the plurality of driving modes including at least thesecond driving mode and the third driving mode is performed bycontrolling steering and acceleration/deceleration of the vehiclewithout depending on an operation performed by the driver of thevehicle, and wherein when a task in the determined driving mode is notperformed by the driver, the driving mode of the vehicle is changed to adriving mode imposing a larger task load; changing the third drivingmode to either one of the first driving mode and the second driving modebased on the operation states when determining based on the operationstates that a failure has occurred in at least one of the plurality ofexternal recognition sensors; changing the third driving mode to thesecond driving mode when determining based on the operation states thatdegradation in performance, which is a phenomenon different from thefailure, has occurred in at least one of the plurality of externalrecognition sensors; and continuing the third driving mode whendetermining that a failure has occurred in at least one of the pluralityof external recognition sensors and the at least one of the plurality ofexternal recognition sensors that has failed is installed on a back sideof the vehicle.
 10. A computer-readable non-transitory storage mediumhaving stored thereon a program for causing a computer mounted on avehicle to: recognize a surrounding situation of a vehicle; controlsteering and acceleration/deceleration of the vehicle based on therecognized surrounding situation without depending on an operationperformed by a driver of the vehicle; detect operation states of aplurality of external recognition sensors; determine a driving mode ofthe vehicle as any one of a plurality of driving modes including a firstdriving mode, a second driving mode, and a third driving mode, whereinthe third driving mode is a driving mode in which a load of a taskimposed on the driver is smaller than a load of a task imposed in thesecond driving mode, wherein the second driving mode is a driving modein which a load of a task imposed on the driver is smaller than a loadof a task imposed in the first driving mode, wherein a part of theplurality of driving modes including at least the second driving modeand the third driving mode is performed by controlling steering andacceleration/deceleration of the vehicle without depending on anoperation performed by the driver of the vehicle, and wherein when atask in the determined driving mode is not performed by the driver, thedriving mode of the vehicle is changed to a driving mode imposing alarger task load; change the third driving mode to either one of thefirst driving mode and the second driving mode based on the operationstates when determining based on the operation states that a failure hasoccurred in at least one of the plurality of external recognitionsensors; change the third driving mode to the second driving mode whendetermining based on the operation states that degradation inperformance, which is a phenomenon different from the failure, hasoccurred in at least one of the plurality of external recognitionsensors; and continue the third driving mode when determining that afailure has occurred in at least one of the plurality of externalrecognition sensors and the at least one of the plurality of externalrecognition sensors that has failed is installed on a back side of thevehicle.